Image search result summarization with informative priors

ABSTRACT

An informative priors image search result summarization system and method that summarizes image search results based on the image relevance (as determined by a search engine&#39;s initial ranking) and the image quality. Embodiments of the system and method cluster the image search results, rank images within each cluster based on a computed image score, and then select a summary image for the cluster. Each cluster is analyzed and an image in the cluster having the maximum image score is included in a selected summary collection. The image score is computed using the image relevance and the image quality, as well as a cluster coherence, a density, and a diversity. The selection of images from a collection of candidate images generates an image search result summarization, which is presented to a user. The summaries are presented to the user in a ranked order based on their image scores.

BACKGROUND

An image search engine provides a convenient tool for users to retrievetheir desired images from the large amount of images on the Web.However, users often find it difficult to identify the interestingimages in the returned results that typically are returned by the searchengine due to the excessive amount of images contained in the returnedresults. One way to lessen the time involved for a user to findinteresting images in the returned results is image search resultsummarization. In general, image search result summarization selectsrepresentative images from the returned results for presentation to theuser and alleviates the need for users to browse each of the images inthe returned results.

For example, consider the situation where a user issues a query “apple”and the search engine returns hundreds of images sorted by relevance.The images returned for query “apple” may range from the fruit apple toApple Inc. products, and even to apple-shaped rock. It is quiteinefficient for a user to browse each image in the returned results tofind the desired images. Actually, when several topics for “apple” arepresented, users are able to obtain their targets more conveniently.

There are a variety of image collection summarization (ICS) techniquesthat are effective in selecting representative images from an imagecollection. In particular, one image collection summarization techniquefor automatically creating image summaries from an image collectionformulates the problem as an optimization problem. This technique takesthe image coverage and diversity into consideration and then describes agreedy algorithm to solve the optimization problem. Another techniqueuses landmark summarization and employs K-Means to cluster the imagesinto visually similar groups. Then the technique select images from theclusters according to some heuristic criteria including visual coherenceand interest point connections. Yet another technique computes anoptimal partition based on a mixture-of-kernels technique and uses asampling algorithm to select representative images. Still anothertechnique uses a greedy method to recommend canonical images. Thistechnique first adopts visual words to represent the visual features ina scene and then iteratively select the images that cover the mostinformative visual words. Another technique clusters photographs byutilizing image content and associated tags to summarize generalqueries, such as “love”, “CLOSEUP” and so on.

Although the above image collection summarization techniques areeffective in selecting representative images from a collection, they arenot optimal to summarize image search results. This is due to severalreasons. One reason is that an image search engine often returns somenoisy images that ideally should not be contained in the summarizationresult. Thus, selecting images primarily by coverage and diversity (asmost of the ICS methods do) is not a good strategy in noisycircumstances. A second reason is that these image collectionsummarization techniques tend to ignore image relevance and assuming theimages in the collection are all relevant. However, the relevanceobtained from the search engine is useful prior information for imagesto be selected as summaries. The third reason is that the image qualityin the summarization result is desirable for a quality users'experience. Low-quality images are non-informative for users if theyoccur in the summaries, since users cannot get the “complex” idea fromsuch small-sized thumbnails. Studies have shown that a user's experiencesignificantly suffers from low-quality summaries and that most userscannot tolerate any thumbnail images with low resolution.

SUMMARY

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter.

Embodiments of the informative priors image search result summarizationsystem and method summarize image search results by taking the relevanceand quality as informative priors. Embodiments of the system and methodcan be broadly divided into two steps: a clustering step and a selectionand ranking step. In the clustering step, embodiments of the system andmethod generate summary candidates based on a clustering method with theinformative priors of relevance and quality. In the selection andranking step, embodiments of the system and method obtain the summary byselecting the top candidates which are ranked according to the prior,reliability and redundancy penalty. Embodiments of the informativepriors image search result summarization system and method can achieve auser friendly summarization in terms of relevance, diversity andcoverage.

Embodiments of the informative priors image search result summarizationsystem and method define image search result summarization as a problemof extracting the most “important” images of search results. Importantimages mean that they are relevant to queries, attractive to users, andrepresentative for the different subtopics. Embodiments of theinformative priors image search result summarization system and methodemploy several criteria (such as relevance and quality) to capturehuman's perception of image summaries. Embodiments of the system andmethod take the initial rank returned by the search engine as therelevance information. Several features including dynamic range, colorentropy, brightness, blur (which describes a sharpness of an image) andcontrast are employed to train a quality model. In order to select therepresentative images with little redundancy, embodiments of theinformative priors image search result summarization system and methodclusters the images to find the exemplars using an affinity propagationtechnique. Embodiments of the system and method then greedily selectsummaries from the exemplars according to several criteria such asrelevance, quality, reliability, and redundancy penalty.

Embodiments of the informative priors image search result summarizationsystem and method input image search results obtained from a searchengine and provide a summarization of the image search results. Theimage search results are initially ranked by the search engine and thenprovided to embodiments of the system and method. Embodiments of thesystem and method then define an image relevance for each image in theimage search results. In some embodiments the image relevance is a rankof an image in the image search results as ranked by the search engine.Embodiments of the system and method then compute an image quality foreach image based on one or more image quality measures, and then clusterimages in the image search results using a clustering technique. Theclustering technique uses as a first informative prior the image qualityfor each image and as a second informative prior the image relevance foreach image. This generates summary candidate collection that containsimage clusters and an exemplar image for each cluster.

Embodiments of the informative priors image search result summarizationsystem and method then select certain images from the summary candidatecollection and then rank the selected images. This generates an imagesearch result summarization, which is presented to a user. The selectionand ranking is achieved by computing an image score for each image inthe summary candidate collection and then ranking each image in thesummary candidate collection based on its image score. In particular,each cluster is analyzed and an image in the cluster having the maximumimage score is selected to be included in a selected summary collection,or image search result summary. The summaries are presented to the userin a ranked order based on image scores.

It should be noted that alternative embodiments are possible, and thatsteps and elements discussed herein may be changed, added, oreliminated, depending on the particular embodiment. These alternativeembodiments include alternative steps and alternative elements that maybe used, and structural changes that may be made, without departing fromthe scope of the invention.

DRAWINGS DESCRIPTION

Referring now to the drawings in which like reference numbers representcorresponding parts throughout:

FIG. 1 is a block diagram illustrating a general overview of embodimentsof the informative priors image search result summarization system andmethod implemented on a computing device.

FIG. 2 is a block diagram illustrating modules and data used inembodiments of the informative priors image search result summarizationsystem and method shown in FIG. 1.

FIG. 3 is a flow diagram illustrating the general operation ofembodiments of the informative priors image search result summarizationsystem shown in FIGS. 1 and 2.

FIG. 4 is a flow diagram illustrating the operational details ofembodiments of the summary candidate generation module shown in FIG. 2.

FIG. 5 is a flow diagram illustrating the operational details ofembodiments of the summary measure computation module shown in FIG. 2.

FIG. 6 is a flow diagram illustrating the operational details ofembodiments of the image summary selection and ranking module shown inFIG. 2.

FIG. 7 illustrates an example of a suitable computing system environmentin which embodiments of the informative priors image search resultsummarization system and method shown in FIGS. 1-6 may be implemented.

DETAILED DESCRIPTION

In the following description of embodiments of the informative priorsimage search result summarization system and method reference is made tothe accompanying drawings, which form a part thereof, and in which isshown by way of illustration a specific example whereby embodiments ofthe informative priors image search result summarization system andmethod may be practiced. It is to be understood that other embodimentsmay be utilized and structural changes may be made without departingfrom the scope of the claimed subject matter.

I. System Overview

FIG. 1 is a block diagram illustrating a general overview of embodimentsof the informative priors image search result summarization system 100and method implemented on a computing device 110. In general,embodiments of the informative priors image search result summarizationsystem 100 and method take an image search result obtained by a searchengine and provide a user with an improved summarization of the results.This image search result summarization is performed taking into accountthe quality of the images in the search result and the relevance (basedon the initial search engine ranking) of the images in the searchresult.

More specifically, embodiments of the informative priors image searchresult summarization system 100 shown in FIG. 1 receive initially-rankedimage search results from a search engine 120. These image searchresults typically are in response to a query from a user to the searchengine. Embodiments of the informative priors image search resultsummarization system 100 then process the initially-ranked image searchresults from the search engine 120 and output a ranked selectedsummaries collection that is displayed to the user 130. This outputsummarizes the search results based on image quality and imagerelevance. In addition, the summaries are ranked so that the user canquickly determine which images returned by the search engine may beclosest to what images the user desired.

FIG. 2 is a block diagram illustrating modules and data used inembodiments of the informative priors image search result summarizationsystem 100 and method shown in FIG. 1. In general, embodiments of theinformative priors image search result summarization system 100 includemodules for generating a collection of candidate images and modules forselecting and ranking images from the candidate collection. Morespecifically, embodiments of the informative priors image search resultsummarization system 100 include a summary candidate generation module200 that generates a collection of candidate images using clusteringtechniques. The summary candidate generation module 200 inputs theinitially-ranked image search results from a search engine 120 and thenprocesses these results. The output of the module is a summary candidatecollection 210, an image quality for each image 220, and an imagerelevance for each image 230. The summary candidate collection 210contains a plurality of images that are clustered based on theclustering techniques. These images are candidates that may be used as asummary image for the search results.

Embodiments of the informative priors image search result summarizationsystem 100 also include an image summary selection and ranking module240 for selecting and ranking images from the image summary collection210. The image summary selection and ranking module 240 includes asummary measure computation module 250 that computes various measures orfeatures of each image. These features include a cluster coherence 260,a density 270, and a diversity 280.

The features are used by the image summary selection and ranking module240 to compute an image score 290 for each image. Each image score alsotakes into account the image quality for each image and the imagerelevance for each image 230. Based on the image score, the imageswithin a cluster are ranked and at least one image from each cluster isselected for inclusion in a ranked selected summaries collection 295.The output is the ranked selected summaries collection that is displayedto the user 130. This display allows the user to quickly find hisdesired images.

II. Operational Overview

FIG. 3 is a flow diagram illustrating the general operation ofembodiments of the informative priors image search result summarizationsystem 100 shown in FIGS. 1 and 2. Referring to FIG. 3, the methodbegins by inputting an initially-ranked image search results from asearch engine (box 300). These are usually images returned by a searchengine in response to an query by a user. The images are ranked by thesearch engine according to relevance.

Next, the system 100 defines an image relevance for each image in theimage search results (box 310). The image relevance is the rank of animage in the initially-ranked image search results, as ranked by thesearch engine. The system 100 then computes an image quality for eachimage (box 320). The image quality may be based upon one or more imagequality measures, as explained in detail below.

The images in the image search results then are clustered using aclustering technique (box 330). The clustering technique uses as a firstinformative prior the image quality for a particular image. Theclustering technique also uses as a second informative prior the imagerelevance for the image. In some embodiments the clustering technique isan Affinity Propagation technique. This clustering generates a summarycandidate collection that contains each of the image contained in thecluster as well as an exemplar image for each cluster.

The system 100 then selects and ranks each image in the summarycandidate collection (box 340). As explained in detail below, thisselecting and ranking is achieved by computing an image score for eachimage using the image quality, image relevance, as well as otherperformance measures. The result of this selecting and ranking is animage search results summarization in the form of a selected summariescollection. This image search results summarization (or selectedsummaries collection) then is presented to a user (box 350). Thesummaries contained in the image search results summarization arepresented in there ranked order, as ranked by the system 100.

III. System and Operational Details

The system and the operational details of embodiments of the informativepriors image search result summarization system 100 and method now willbe discussed. These embodiments include embodiments of the summarycandidate generation module 200, the summary measure computation module250, and the image summary selection and ranking module 24. The systemand operational details of each of these modules now will be discussedin detail.

III.A. Summary Candidate Generation Module

The summary candidate generation module 200 generates a summary ofcandidates images based on image quality and relevance. For each imagein the initial search results, the module 200 first estimate therelevance and quality, which then is combined as a prior as to whetherthe image is selected as a summary. The module 200 then selectsexemplars by using a clustering technique (such as an AffinityPropagation (AP) algorithm) with the prior to generate the candidatesfor summarization.

III.A.1. Affinity Propagation Algorithm Overview

The summary candidate generation module 200 uses an exemplar-basedclustering algorithm. In some embodiments, this exemplar-basedclustering algorithm is an Affinity Propagation (AP) algorithm. Thereason for using the AP clustering method is two-fold. First, it isdifficult for other clustering methods to take the relevance and qualityfactors into account as a prior, while the AP algorithm allows a priorto be assigned for each image. Second, the AP clustering method does notrequire predefining the number of clusters, which is usually hard todetermine for the summarization problem.

A general mathematical description of the AP clustering method is asfollows. Considering all N data points as potential exemplars, the APalgorithm clusters data according to two kinds of messages exchangedbetween data points. One kind of message is the “responsibility” r(i,k),sent from data point i to k. This term reflects how well-suited k is toserve as the exemplar for i in view of other potential exemplars. Thekind of messages is the “availability” a(i,k). This is sent from point kto i and reflects how appropriate it would be for i to choose k as itsexemplar considering the support from other points that k is a candidatefor an exemplar. The computational cost of the AP algorithm is O(N²T)where T is the number of iterations.

One input to the AP algorithm is the similarity matrix of the N datapoints. Another input is the preference, which can be regarded as theinformative prior for each image to be selected as an exemplar. With theinformative prior, the AP algorithm does not need to specify the numberof clusters. In the output of AP clustering algorithm, every data pointi has its corresponding exemplar k. This means that the image I_(i) canbe represented by I_(k). This is denoted as S(I_(i))=I_(k).

III.A.2. Using Image Quality and Relevance as Measures in the APAlgorithm

The summary candidate generation module 200 uses several criteria tomeasure the prior of an image to be contained in a summary. Each priorfor an image then is used by the AP framework. Mathematically, theinformative prior for each image as an exemplar is estimated using alinear model of the relevance R(I_(i),q) and quality Q(I_(i)) given by:Prior(I _(i) ,q)=ω₁ R(I _(i) ,q)+ω₂ Q(I _(i))+c  1where I_(i) is the i^(th) image in the search result, q is a givenquery, and c is a constant.

In the following sections the estimation of relevance and quality is setforth.

III.A.2.i. Relevance

In general, relevance is estimated by making use of the initial rank ofan image. The initial rank records the ranked position of each imagereturned directly by the search engine. This initial ranking is usefulbecause top-ranked images are more likely to be representative imagesthan bottom-ranked images. This suggests that relevance is a factor thatinfluences a human's decision to select summary images and that the rankprovided by search engines is a good indication of the “true” relevanceof an image.

Mathematically, given N retrieved images under a specified query q, therelevance score for each image I_(i) is defined as:R(I _(i) ,q)=1−Pos(I _(i) ,q)/N,i=1, . . . , N  2where Pos(I_(i),q) is the position of the image I_(i) in the searchresult.III.A.2.ii. Quality

Images presenting a good appearance are likely to attract moreattention. Good appearance means that the image has both a clear viewand high aesthetics. The system and method use the following set ofquality measures that are effective in describing the quality of animage to predict whether an image has a good appearance. In variousembodiments, a single quality measure, any combination of a qualitymeasures, or all of the quality measures, may be used by the summarycandidate generation module 200 to predict image quality.

Dynamic Range: Dynamic range is used for denoting the luminance range ofa scene being photographed. The value is computed by the ratio betweenthe maximum and minimum measurable light intensities.

Color Entropy: Color entropy may be used to describe the colorlessnessof the image content.

Brightness: A large amount of low-quality images are photographed withinsufficient light. Any one of a number of available algorithms can beused to calculate the brightness for each image.

Blur: Any one of a number of blur algorithms can be used to find theblur. A blur algorithm that is designed to work well for web images isuseful.

Contrast: Good images are generally under strong contrast between thesubject and the background. A number of available algorithms can be usedcompute the contrast.

Each of the above-described quality measures returns a score for each ofthe images. The quality factor Q(I_(i)) is further a linear combinationof dynamic range, color entropy, brightness, blur and contrast. To learnthe weights of the quality factors automatically, the system and methodconstruct a training set by labeling images into low-quality images(which are fuzzy and unpleasant images), middle-quality images (whichare not good enough to be contained in a summary), and high-qualityimages (good looking and easy to understand). In some embodiments aranking support vector machine technique is used to train the qualitymodel.

III.A.2.iii. AP Algorithm Incorporating Image Quality and Relevance

FIG. 4 is a flow diagram illustrating the operational details ofembodiments of the summary candidate generation module 200 shown in FIG.2. The operation begins by obtaining a plurality of initially-rankedimages (box 400). These images are the images returned by a searchengine in response to a user's search query, and are ranked by thesearch engine accordingly to relevance to the user's search query. Next,the module 200 sets as a first preference (or informative prior) therelevance (or initial ranking) of each image contained in the resultsreturned by the search engine (box 410).

One or more image quality measures also are selected (box 420). Thisselection may be done automatically by the module 200 or selectedmanually (such as by the user). Next, the module 200 computes an imagequality of each image in the plurality of initially-ranked images usingthe selected image quality measures (box 430). The module 200 then setsas a second preference (or informative prior) the image quality of eachimage in the plurality of initially-ranked images (box 440).

A clustering technique then is used to cluster the plurality ofinitially-ranked images (box 450). The clustering technique clusters theimages based on the image relevance and the image quality of each image.In some embodiments, the AP algorithm is used as the clusteringtechnique. Mathematically, the AP algorithm clusters the top N imagesand outputs selected exemplars. The AP algorithm uses the image initialrank (which is the position in the returned result by the search engine)and image quality and sets them as the preferences (or informativepriors). In some embodiments the image quality is accessed using colorentropy

Denoting the image in the i^(th) position in the search result as I_(i),the quality score of image I_(i) is Q(i) and R(i) is a transformation toobtain a relevance score from the ranking position, and then thepreference (or informative prior) is estimated asP _(i) =α×R(i)+β×Q(i)+cThe exemplars selected are denoted as E. As explained in detail below,since the number of exemplars may be larger than a desired number ofsummaries, a post-selection step may be performed to select the desiredimages from the exemplars for summarization.

The result of this clustering is a plurality of clusters. The module 200then selects an exemplar from each cluster (box 460). The exemplar is animage that represents the cluster. Each exemplar then is saved in asummary candidate collection. This collection contains candidates thatmay be used in the final summarization of the image search result. Theoutput of the module 200 is the image quality and image relevance foreach image (box 470), and the summary candidate collection that containsboth the images in each cluster and the selected exemplars for eachcluster (480).

III.B. Image Summary Selection and Ranking Module

The image summary selection and ranking module 240 both ranks andselects the most competitive summary candidates to form a summary.Competitiveness is a measure that serves to minimize the redundancywhile maximizing both the candidate's prior confidence and reliability.

III.B.1. Summary Measure Computation Module

The image summary selection and ranking module 240 includes a summarymeasure computation module 250. FIG. 5 is a flow diagram illustratingthe operational details of embodiments of the summary measurecomputation module 250 shown in FIG. 2. In general, the module 250computes competitiveness measures that are to compute a score for eachimage. These image scores are used to select summary images for eachcluster in the summary candidate collection and then rank those summaryimages for presentation to a user.

More specifically, the operation of the module 250 begins by inputting acluster from the summary candidate (box 500). Next, the module 250selects an image from the cluster (box 505). In general, each image fromeach cluster is processed in the following manner. First, the module 250computes a visual distance between the selected image and other imagesin the selected cluster (box 510). Next, a scaling parameter is defined(box 515), and the module 250 computes a similarity between the selectedimage and the other images in the cluster using the visual distance andthe scaling parameter (box 520).

The module 250 then computes (box 525) a cluster coherence 530 for theselected cluster. Moreover, the module 250 computes (box 535) a density540, and computes (box 545) a diversity 550 for the selected image usingthe computed similarity. The cluster coherence 530 is saved for theselected cluster, while the density 540, and diversity 550 are saved forthe selected image (box 555).

The cluster coherence 530 is computed as the sum of the similaritiesbetween the all of the pairs of images in the cluster. For the clusters,where all of the users are close, the coherence is good. For clusterswhere the images are farther away the coherence is not as good. A largercluster coherence 530 means the images pairs are more similar ascompared to a smaller cluster coherence 530. This means that the imagespairs are less similar. It should be noted that the cluster coherence530 measures the quality of the cluster (and not the individual images).

The second measure is the density 540, which is the probability that theimage will be in the search results. If there are many similar images inthe results, then this corresponds to a higher density 540. The thirdmeasure is the diversity 550 function, which determines the similarityof the image to the selected images. The idea is that for 4 selectedimages, if the fifth image is similar to any of the previous 4 images,then it should not be selected because it has low diversity 550 andinformation. This is to avoid redundancy in the selected image. In otherwords, if 4 images are selected and it is only similar to one of the 4picked images, then this is a smaller diversity 550. The goal is to tryand make the selected images not redundant. Note that If diversity 550is high then redundancy is low. It should also be noted that thecomputation of the cluster coherence 530, density 540, and diversity 550can be done simultaneously.

Mathematically, in some embodiments the visual distance and similarityare computed as follows. The distance between two images Dis(i,j) iscomputed using standard techniques. Then, the similarity S(I_(i), I_(j))is based on the distance.

${S\left( {I_{i},I_{j}} \right)} = {\exp\left( {- \frac{{Dis}\left( {i,j} \right)}{\sigma^{2}}} \right)}$where σ is the scaling parameter.

In some embodiments the cluster coherence 530 is defined as follows. Foreach exemplar image E_(i), there is a cluster of images associated withit as outputted in the clustering technique (such as the AffinityPropagation algorithm, which is defined as C_(i). Then the clustercoherence 530 of the cluster C_(i) is defined as:

${{Coh}(i)} = {\sum\limits_{I_{i},{I_{j} \in C_{i}}}\;{S\left( {I_{i},I_{j}} \right)}}$where S(I_(i), I_(j)) is the similarity between the two images.

In some embodiments the density 540 is estimated using a classicalmethod, the Kernel Density Estimation, as follows.

${{{Dens}(i)} = {\sum\limits_{I_{j}}{S\left( {I_{i},I_{j}} \right)}}},$where I_(j) are the images in the search result other than I_(i).

In some embodiments the diversity 550 is computed as follows. Define theselected images as B. Then the diversity 550 measure for image I_(i) isdefined as:Div(i)=−max_(I) _(jεB) S(I _(i) ,I _(j)).

The module 250 then determines whether there are more images in theselected cluster to process (box 560). If so, then the module 250selects another image from the selected cluster (box 565), and theprocess begins again for the new selected image. Otherwise, the module250 outputs the cluster coherence, density, and diversity for each imagein the summary candidate collection (box 570).

III.B.2. Operation of the Image Summary Selection and Ranking Module

FIG. 6 is a flow diagram illustrating the operational details ofembodiments of the image summary selection and ranking module 240 shownin FIG. 2. The operation begins by inputting the summary candidatecollection along with the image from the initial search results (box600). Next, a desired number of summaries, N_(S), is selected (box 605).In some embodiments the selection is automatically done by the module240, while in other embodiments the selection is done manually by auser.

The module 240 then defines a selected summaries collection, A, that isinitially an empty (or null) set (box 610). This initialization actionsets the select summaries collection, to an empty set. Moreover, thesummary candidate collection is denoted as CAN, which is the images andcluster generated by the clustering technique of the summary candidategeneration module 200.

The module 240 then select a cluster from the summary candidatecollection (box 615). In addition, an image from the selected cluster isselected by the module 240 (box 620). A determination then is made as towhether the number of images in the selected summaries collection isless than the desired number of summaries (box 625). If so, then themodule 240 sets the summary candidate collection equal to the image inthe initial search results minus the image in the summary candidatecollection (box 630).

Otherwise, the module 240 makes a determination as to whether the numberof images in the summary candidate collection is greater than zero (box635). If so, then the module 240 obtains the cluster coherence 530, thedensity 540, the diversity 550, the image quality, and the imagerelevance for the selected image (box 640). These measure are obtainedfrom the summary candidate generation module 200 and the image summaryselection and ranking module 240.

Next, the module 240 computes an image score for the selected imageusing the cluster coherence 530, the density 540, the diversity 550, theimage quality, and the image relevance for the selected image (box 645).Mathematically, the image score, S_(i) for each of the images in summarycandidate collection, CAN, is computed as:

S_(i) = W₁ × Coh(i) + W₂ × Dens(i) + W₃ × Div(i) + α × R(i) + β × Q(i),Where Coh(i) is the cluster coherence 530 for a selected cluster,Dens(i) is the density 540 for the i^(th) image, Div(i) is the diversity550 for the i^(th) image, R(i) is the image relevance for the i^(th)image, and Q(i) is the image quality for the i^(th) image. Moreover, W₁is a first weight, which is the weight of the cluster coherence 530term, W₂ is a second weight, which is the weight of the density 540term, and W₃ is a third weight, which is the weight of the diversity 550term. For example, W₁ would be larger that the other weights if it wasdesirable to give more weight to the coherence 530 term. In someembodiments, the weights are determined either by the user and in otherembodiments the weights are determined based on experimental results ofwhich combination of weights provides the most desirable results.Moreover, α is a first parameter, which is a parameter for the imagerelevance, and β is a second parameter, which is a parameter for theimage quality.

The image score for the selected image is added to a list of imagesscore for the selected cluster (box 650). A determination then is madeas to whether the cluster contains more images (box 655). If so, thenthe module 240 selects another image from the selected cluster (box 660)and the process is performed on the new selected image. If not, then themodule 240 searches the list of images scores to find and identify theimage in the cluster having a maximum image score for the cluster (box665).

The module 240 then adds the image in the selected cluster having themaximum image score to the selected summaries collection (box 670).Moreover, the module 240 removes the images having the maximum imagescore from the summary candidate collection (box 675). A determinationthen is made as to whether there are more clusters to be processed (box680). If so, then the module 240 selects another cluster (box 685) andprocesses each of the images in the cluster as set forth above. Theimage in each cluster having the maximum image score is added to theselected summaries collection. If there are no more clusters, or if thenumber of images in the selected summaries collection is greater thanthe desired number of summaries (box 625), then the module 240 displaysthe selected summaries collection to a user (box 690). The selectedsummaries collection is displayed to the user in a ranked manner, wherethe ranking is based on the image score associated with each image.

IV. Exemplary Operating Environment

Embodiments of the informative priors image search result summarizationsystem 100 and method are designed to operate in a computingenvironment. The following discussion is intended to provide a brief,general description of a suitable computing environment in whichembodiments of the informative priors image search result summarizationsystem 100 and method may be implemented.

FIG. 7 illustrates an example of a suitable computing system environmentin which embodiments of the informative priors image search resultsummarization system 100 and method shown in FIGS. 1-6 may beimplemented. The computing system environment 700 is only one example ofa suitable computing environment and is not intended to suggest anylimitation as to the scope of use or functionality of the invention.Neither should the computing environment 700 be interpreted as havingany dependency or requirement relating to any one or combination ofcomponents illustrated in the exemplary operating environment.

Embodiments of the informative priors image search result summarizationsystem 100 and method are operational with numerous other generalpurpose or special purpose computing system environments orconfigurations. Examples of well known computing systems, environments,and/or configurations that may be suitable for use with embodiments ofthe informative priors image search result summarization system 100 andmethod include, but are not limited to, personal computers, servercomputers, hand-held (including smartphones), laptop or mobile computeror communications devices such as cell phones and PDA's, multiprocessorsystems, microprocessor-based systems, set top boxes, programmableconsumer electronics, network PCs, minicomputers, mainframe computers,distributed computing environments that include any of the above systemsor devices, and the like.

Embodiments of the informative priors image search result summarizationsystem 100 and method may be described in the general context ofcomputer-executable instructions, such as program modules, beingexecuted by a computer. Generally, program modules include routines,programs, objects, components, data structures, etc., that performparticular tasks or implement particular abstract data types.Embodiments of the informative priors image search result summarizationsystem 100 and method may also be practiced in distributed computingenvironments where tasks are performed by remote processing devices thatare linked through a communications network. In a distributed computingenvironment, program modules may be located in both local and remotecomputer storage media including memory storage devices. With referenceto FIG. 7, an exemplary system for embodiments of the informative priorsimage search result summarization system 100 and method includes ageneral-purpose computing device in the form of a computer 710.

Components of the computer 710 may include, but are not limited to, aprocessing unit 720 (such as a central processing unit, CPU), a systemmemory 730, and a system bus 721 that couples various system componentsincluding the system memory to the processing unit 720. The system bus721 may be any of several types of bus structures including a memory busor memory controller, a peripheral bus, and a local bus using any of avariety of bus architectures. By way of example, and not limitation,such architectures include Industry Standard Architecture (ISA) bus,Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, VideoElectronics Standards Association (VESA) local bus, and PeripheralComponent Interconnect (PCI) bus also known as Mezzanine bus.

The computer 710 typically includes a variety of computer readablemedia. Computer readable media can be any available media that can beaccessed by the computer 710 and includes both volatile and nonvolatilemedia, removable and non-removable media. By way of example, and notlimitation, computer readable media may comprise computer storage mediaand communication media. Computer storage media includes volatile andnonvolatile removable and non-removable media implemented in any methodor technology for storage of information such as computer readableinstructions, data structures, program modules or other data.

Computer storage media includes, but is not limited to, RAM, ROM,EEPROM, flash memory or other memory technology, CD-ROM, digitalversatile disks (DVD) or other optical disk storage, magnetic cassettes,magnetic tape, magnetic disk storage or other magnetic storage devices,or any other medium which can be used to store the desired informationand which can be accessed by the computer 710. By way of example, andnot limitation, communication media includes wired media such as a wirednetwork or direct-wired connection, and wireless media such as acoustic,RF, infrared and other wireless media. Combinations of any of the aboveshould also be included within the scope of computer readable media.

The system memory 730 includes computer storage media in the form ofvolatile and/or nonvolatile memory such as read only memory (ROM) 731and random access memory (RAM) 732. A basic input/output system 733(BIOS), containing the basic routines that help to transfer informationbetween elements within the computer 710, such as during start-up, istypically stored in ROM 731. RAM 732 typically contains data and/orprogram modules that are immediately accessible to and/or presentlybeing operated on by processing unit 720. By way of example, and notlimitation, FIG. 7 illustrates operating system 734, applicationprograms 735, other program modules 736, and program data 737.

The computer 710 may also include other removable/non-removable,volatile/nonvolatile computer storage media. By way of example only,FIG. 7 illustrates a hard disk drive 741 that reads from or writes tonon-removable, nonvolatile magnetic media, a magnetic disk drive 751that reads from or writes to a removable, nonvolatile magnetic disk 752,and an optical disk drive 755 that reads from or writes to a removable,nonvolatile optical disk 756 such as a CD ROM or other optical media.

Other removable/non-removable, volatile/nonvolatile computer storagemedia that can be used in the exemplary operating environment include,but are not limited to, magnetic tape cassettes, flash memory cards,digital versatile disks, digital video tape, solid state RAM, solidstate ROM, and the like. The hard disk drive 741 is typically connectedto the system bus 721 through a non-removable memory interface such asinterface 740, and magnetic disk drive 751 and optical disk drive 755are typically connected to the system bus 721 by a removable memoryinterface, such as interface 750.

The drives and their associated computer storage media discussed aboveand illustrated in FIG. 7, provide storage of computer readableinstructions, data structures, program modules and other data for thecomputer 710. In FIG. 7, for example, hard disk drive 741 is illustratedas storing operating system 744, application programs 745, other programmodules 746, and program data 747. Note that these components can eitherbe the same as or different from operating system 734, applicationprograms 735, other program modules 736, and program data 737. Operatingsystem 744, application programs 745, other program modules 746, andprogram data 747 are given different numbers here to illustrate that, ata minimum, they are different copies. A user may enter commands andinformation (or data) into the computer 710 through input devices suchas a keyboard 762, pointing device 761, commonly referred to as a mouse,trackball or touch pad, and a touch panel or touch screen (not shown).

Other input devices (not shown) may include a microphone, joystick, gamepad, satellite dish, scanner, radio receiver, or a television orbroadcast video receiver, or the like. These and other input devices areoften connected to the processing unit 720 through a user inputinterface 760 that is coupled to the system bus 721, but may beconnected by other interface and bus structures, such as, for example, aparallel port, game port or a universal serial bus (USB). A monitor 791or other type of display device is also connected to the system bus 721via an interface, such as a video interface 790. In addition to themonitor, computers may also include other peripheral output devices suchas speakers 797 and printer 796, which may be connected through anoutput peripheral interface 795.

The computer 710 may operate in a networked environment using logicalconnections to one or more remote computers, such as a remote computer780. The remote computer 780 may be a personal computer, a server, arouter, a network PC, a peer device or other common network node, andtypically includes many or all of the elements described above relativeto the computer 710, although only a memory storage device 781 has beenillustrated in FIG. 7. The logical connections depicted in FIG. 7include a local area network (LAN) 771 and a wide area network (WAN)773, but may also include other networks. Such networking environmentsare commonplace in offices, enterprise-wide computer networks, intranetsand the Internet.

When used in a LAN networking environment, the computer 710 is connectedto the LAN 771 through a network interface or adapter 770. When used ina WAN networking environment, the computer 710 typically includes amodem 772 or other means for establishing communications over the WAN773, such as the Internet. The modem 772, which may be internal orexternal, may be connected to the system bus 721 via the user inputinterface 760, or other appropriate mechanism. In a networkedenvironment, program modules depicted relative to the computer 710, orportions thereof, may be stored in the remote memory storage device. Byway of example, and not limitation, FIG. 7 illustrates remoteapplication programs 785 as residing on memory device 781. It will beappreciated that the network connections shown are exemplary and othermeans of establishing a communications link between the computers may beused.

The foregoing Detailed Description has been presented for the purposesof illustration and description. Many modifications and variations arepossible in light of the above teaching. It is not intended to beexhaustive or to limit the subject matter described herein to theprecise form disclosed. Although the subject matter has been describedin language specific to structural features and/or methodological acts,it is to be understood that the subject matter defined in the appendedclaims is not necessarily limited to the specific features or actsdescribed above. Rather, the specific features and acts described aboveare disclosed as example forms of implementing the claims appendedhereto.

1. A method implemented on a computing device having a processor forsummarizing image search results, comprising: using the computing devicehaving the processor to perform the following: estimating an imagerelevance for each image in the image search results as a ranking ofeach image given by a search engine that provided the image searchresults in response to a query from a user to the search engine;computing an image quality for each image based on one or more imagequality measures; clustering images in the image search results using aclustering technique that has as a first informative prior the imagequality for each image and as a second informative prior the imagerelevance for each image to obtain a summary candidate collectioncontaining image clusters and an exemplar image for each cluster;selecting and ranking each image in the summary candidate collection toobtain an image search results summarization; selecting a numberrepresenting a desired number of summaries; and presenting the imagesearch results summarization to a user based on whether a number ofimages contained in the image search results summarization is less thanthe number representing a desired number of summaries.
 2. The method ofclaim 1, further comprising: computing an image score for each image inthe summary candidate collection; and ranking each image in the summarycandidate collection based on its image score.
 3. The method of claim 2,further comprising: computing a similarity between a selected image andanother image in a image cluster; using the similarity to compute acluster coherence, density, and diversity for the selected image.
 4. Themethod of claim 3, further comprising computing an image score for theselected image using the cluster coherence, density, diversity, imagequality, and image relevance.
 5. The method of claim 4, furthercomprising: computing an image score for each image in a the imagecluster; and identifying an image in the image cluster having a maximumimage score.
 6. The method of claim 5, further comprising removing theimage in the image cluster having the maximum image score from thesummary candidate collection: adding the image in the image clusterhaving the maximum image score to the selected summaries collection; andremoving the image in the image cluster having the maximum image scorefrom the summary candidate collection.
 7. The method of claim 6, furthercomprising presenting images in the selected summaries collection to theuser in an order according to a ranking of the images based on the imagescore of each image.
 8. The method of claim 1, further comprisingcomputing the image quality for each image using color entropy of theimage as an image quality measure.
 9. The method of claim 1, furthercomprising clustering images in the image search results using anAffinity Propagation clustering technique that has as a first preferencethe image quality and as a second preference the image relevance foreach image.
 10. The method of claim 9, further comprising computing theimage quality for each image using at least one of the following as animage quality measure: (a) dynamic range, which denotes a luminancerange of a scene being photographed and is a ratio between a maximum anda minimum measurable light intensities; (b) color entropy, whichdescribes a colorlessness of image content; (c) brightness, whichdescribes an amount of light in an image; (d) blur, which describes asharpness of an image; and, (e) contrast, where good-quality images aregenerally under strong contrast between a subject and a background. 11.A method implemented on a computing device having a processor forperforming image search results summarization on a plurality ofinitially-ranked search results ranked by a search engine, comprising:using the computing device having the processor to perform thefollowing: setting as a first preference an image relevance for eachimage corresponding to an initial ranking by the search engine;selecting one or more image quality measures; computing an image qualityfor each image using the selected image quality measures; clusteringimages in the plurality of initially-ranked search results using aclustering algorithm using the first preference and a second preferenceto obtain a plurality of clusters; selecting an exemplar image for eachof the plurality of clusters and including the exemplar image and imagesin the plurality of clusters in a summary candidate collection;selecting a cluster and an image from the summary candidate collection;selecting a number representing a desired number of summaries;determining whether a number of images contained in a selected summariescollection is less than the number representing a desired number ofsummaries; if so, then setting the summary candidate collection equal toimages in the plurality of initially-ranked search results minus thenumber of images contained in the selected summaries collection;computing an image score for each image in the selected cluster usingthe image relevance, the image quality, a cluster coherence, a density,and a diversity; identifying an image in the selected cluster having amaximum image score as compared to other images in the selected cluster;adding the image having a maximum image score to a selected summariescollection and removing the image having a maximum image score from thesummary candidate collection; and displaying to a user images in theselected summary collection in a ranked order based on the image scoreof the image.
 12. The method of claim 11, further comprising:determining whether a number of images contained in the summarycandidate collection is greater than zero; and if not, then displayingto the user the images in the selected summaries collection.
 13. Themethod of claim 11, further comprising: computing a visual distancebetween a selected image and other images in the selected cluster;defining a scaling parameter; computing a similarity between theselected image and the other images in the selected cluster using thevisual distance and the scaling parameter.
 14. The method of claim 13,further comprising computing the cluster coherence, density, anddiversity for the selected image using the computed similarity.
 15. Acomputer-implemented method for generating summary images for an imagesearch result containing a plurality of initially-ranked images,comprising: setting as a first preference an image relevance of theplurality of initially-ranked images, denoted as R(i), which is an imagerelevance of an image in the i^(th) position of the image search result;computing an image quality of each of the plurality of initially-rankedimages using a quality measure based on color entropy, denoted as Q(i),which is an image quality of an image in the i^(th) position of theimage search result; clustering the plurality of initially-ranked imagesusing an Affinity Propagation clustering technique having the imagerelevance as the first preference and the image quality as the secondpreference to obtain a plurality of clusters; selecting an exemplarimage from each of the plurality of clusters and including the exemplarsand images in the plurality of clusters in a summary candidatecollection; selecting a cluster from the summary candidate collectionand an image from the selected cluster; obtaining a cluster coherence,Coh(i), a density, Dens(i), a diversity, Div(i), the image quality,Q(i), and the image relevance, R(i), for an i^(th) image in the selectedcluster; computing an image score, S_(i), for the i^(th) image using thefollowing equation:S _(i) =W ₁×Coh(i)+W ₂×Dens(i)+W ₃×Div(i)+α×R(i)+β×Q(i), where W₁ is afirst weight, W₂ is a second weight, W₃ is a third weight, α is a firstparameter, and β is a second parameter, until each image in the selectedcluster has an image score; identifying an image in the selected clusterhaving a maximum image score, removing the identified image from thesummary candidate collection, and adding the identified image to aselected summaries collection; and displaying to a user images in theselected summaries collection that are ranked accordingly to arespective image score.
 16. The computer-implemented method of claim 15,further comprising clustering the plurality of initially-ranked imagesusing the Affinity Propagation clustering technique to find an overallpreference, P(i), for the image in the i^(th) position of the imagesearch result using the equation:P(i)=α×R(i)+β×Q(i)+c, where c is a constant.
 17. Thecomputer-implemented method of claim 15, further comprising computingthe cluster coherence, Coh(i), for an i^(th) image in the selectedcluster, C_(i), using the equation:Coh(i)=Σ_(I) _(i) _(,I) _(j) _(εC) _(i) S(I _(i) ,I _(j)), whereS(I_(i),I_(j)) is a similarity based on a distance between the i^(th)image, I_(i), and a j^(th) image, I_(j), and the distance, Dis(i,j),given by the equation:${S\left( {I_{i},I_{j}} \right)} = {{\exp\left( {- \frac{{Dis}\left( {i,j} \right)}{\alpha^{2}}} \right)}.}$18. The computer-implemented method of claim 17, further comprisingcomputing the density, Dens(i), for the i^(th) image in the selectedcluster, using the equation:Dens(i)=Σ_(I) _(j) S(I _(i) ,I _(j)), where I_(j) is an image in theimage search result other than I_(i).
 19. The computer-implementedmethod of claim 18, further comprising computing the diversity, Div(i),for the i^(th) image in the selected cluster, using the equation:Div(i)=max_(I) _(j) _(εA) S(I _(i) ,I _(j)), where A is a set ofselected images.